Multi-device coordinated user experience

ABSTRACT

Techniques are described for providing a simultaneous, multi-device user experience for one or more users, on computing devices that include different capabilities for data input and/or data output. A user may register multiple devices with a service, and the service may determine capabilities of the devices for data input and data output. During a communication session involving the user and another user, such as a service representative, the service may receive input data collected through one or more devices and/or provide output data to be presented on one or more devices, where such data input and data output is received and provided according to the various input/output capabilities of the various devices. The data input and/or data output may switch from device to device during a session. In some instances, data may be replicated for presentation through multiple devices simultaneously.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation application of and claims priorityunder 35 U.S.C. § 120 to U.S. application Ser. No. 15/192,637, filed onJun. 24, 2016, the entire contents of which is incorporated herein byreference.

BACKGROUND

In recent years, various ecosystems of network-connected smartappliances, Internet of Things (IoT) devices, mobile devices, and/orother computing devices have become available. Such devices may havevarying capabilities with regard to data input and data output. Forexample, a device may accept audio input (e.g., speech input) andprovide audio output, but may not include a display for visuallypresenting data. As another example, a device such as a television mayinclude a large display but may lack a full-featured user interface forinputting data.

SUMMARY

Implementations of the present disclosure are generally directed toproviding a coordinated user experience across multiple devices. Morespecifically, implementations are directed to providing a coordinateduser experience across multiple devices with varying input and/or outputcapabilities, the user experience provided during a service session inwhich a service representative interacts with one or more users usingthe input and/or output capabilities of the multiple devices.

In general, innovative aspects of the subject matter described in thisspecification can be embodied in methods that include actions of:determining a first client device to be used for collecting firstcontent during a session, the first client device being determinedduring the session based at least partly on the first client deviceincluding a particular data input capability for collecting the firstcontent; receiving the first content that is collected through the firstclient device during the session; determining a second client device tobe used for presenting second content during the session, the secondclient device being determined during the session based at least partlyon the second client device including a particular data outputcapability for presenting the second content, the second client devicebeing different than the first client device; and providing the secondcontent for presentation through the second client device during thesession.

Implementations can optionally include one or more of the followingfeatures: the first content is provided through the first client deviceby a first user; the second content is presented through the secondclient device to a second user; the first client device is in adifferent geographic location from the second client device; thedetermining of the second client device is further based on a currentcontext of the second client device, including one or more of a locationor a state of the second client device; the determining of the secondclient device is further based on an indication of at least one userparticipating in the session; the first content includes audio inputprovided by a first user participating in the session; the secondcontent includes visual output provided by a second user participatingin the session; the second content is determined based at least partlyon the first content; the actions further include determining a thirdclient device to be used for presenting third content during thesession, the third client device being determined during the sessionbased at least partly on the third client device including a particulardata output capability for presenting the third content, the thirdclient device being different than the first client device and thesecond client device; the actions further include providing the thirdcontent for presentation through the third client device during thesession; at least one of the second content or the third content isreplicated for presentation through both the second client device andthe third client device during the session; and/or the actions furtherinclude registering the first client device and the second clientdevice, the registering including determining and storing a set ofcapabilities of each of the first client device and the second clientdevice, the set of capabilities including one or more of a data outputcapability or a data input capability.

Other implementations of any of the above aspects include correspondingsystems, apparatus, and computer programs that are configured to performthe actions of the methods, encoded on computer storage devices. Thepresent disclosure also provides a computer-readable storage mediumcoupled to one or more processors and having instructions stored thereonwhich, when executed by the one or more processors, cause the one ormore processors to perform operations in accordance with implementationsof the methods provided herein. The present disclosure further providesa system for implementing the methods provided herein. The systemincludes one or more processors, and a computer-readable storage mediumcoupled to the one or more processors having instructions stored thereonwhich, when executed by the one or more processors, cause the one ormore processors to perform operations in accordance with implementationsof the methods provided herein.

Implementations of the present disclosure provide one or more of thefollowing advantages. Traditionally, a session in which one user (e.g.,a customer) consults with another user (e.g., a service representative)would be conducted using a single device, e.g., through a telephone callor a customer service web site. However, the single device may havelimited capabilities for data input and/or data output, leading topossible confusion, miscommunication, repetition, and/or otherinefficiencies in conducting the session. By leveraging the variouscapabilities of the various computing devices that may be present in theuser's environment, implementations provide for an enriched userexperience for each of the individuals participating in the sessioncompared to a traditional interaction using a single device. Through theenriched user experience, the session may proceed more quickly with alower likelihood of confusion, miscommunication, or other problems.Accordingly, the multi-device session provided by implementationsconsumes less network usage time, processing capacity, memory, and/orother computing resources compared to a traditional single-devicesession.

Moreover, the simultaneous multi-device and/or multi-user experienceprovided by implementations may provide advantages in scenarios where ana service representative of an organization is interacting with a user(e.g., customer), as well as scenarios where multiple users (e.g.,spouses) are participating in a same service session. In the latterscenario, the users may be geographically remote from one another. Forexample, one spouse may be deployed in the military, travelling forwork, and so forth. In general, implementations may provide amulti-device and multi-user experience that includes a servicerepresentative and any suitable number of other users who are beingassisted during a session.

It is appreciated that aspects and features in accordance with thepresent disclosure can include any combination of the aspects andfeatures described herein. That is, aspects and features in accordancewith the present disclosure are not limited to the combinations ofaspects and features specifically described herein, but also include anycombination of the aspects and features provided.

The details of one or more implementations of the present disclosure areset forth in the accompanying drawings and the description below. Otherfeatures and advantages of the present disclosure will be apparent fromthe description and drawings, and from the claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 depicts an example system for providing a multi-devicecoordinated user experience, according to implementations of the presentdisclosure.

FIG. 2 depicts an example session conducted through a multi-devicecoordinated user experience, according to implementations of the presentdisclosure.

FIG. 3 depicts a flow diagram of an example process for registering aclient device, according to implementations of the present disclosure.

FIG. 4 depicts a flow diagram of an example process for providing amulti-device coordinated user experience, according to implementationsof the present disclosure.

FIG. 5 depicts an example computing system, according to implementationsof the present disclosure.

DETAILED DESCRIPTION

Implementations of the present disclosure are directed to systems,devices, methods, and computer-readable media for providing asimultaneous, multi-device user experience for one or more users, oncomputing devices that include different capabilities for data inputand/or data output. A user may register multiple devices with a service,and the service may determine capabilities of the devices for data inputand data output. For example, the service may determine that a headlesspersonal assistant device is able to accept speech input and provideaudio output, but does not have a display. As another example, theservice may determine that a smart television has a large display but alimited (or no) user interface. During a communication session involvingthe user and another user, such as a service representative (SR), theservice may receive input data collected through one or more devicesand/or provide output data to be presented on one or more devices, wheresuch data input and data output is received and provided according tothe various input/output capabilities of the various devices. The datainput and/or data output may switch from device to device during asession. In some instances, data may be replicated for presentationthrough multiple devices simultaneously, e.g., in instances wheremultiple, geographically remote users are participating in a sessionusing different, geographically separate client devices.

A user may own and/or operate various network-connected computingdevices in their home, office, and/or other environments. Each of thevarious computing device(s) may have its own set of capabilities withregard to data input and data output to assist a user in understandingand/or controlling their environment. For example, different devices mayhave different sizes and/or types of displays, or may lack a display.Different devices may have different capabilities with regard to audioinput and/or audio output, or may lack such capabilities. Differentdevices may include different types of user interfaces for text input,gestural input, interacting with graphical user interface (GUI)elements, and so forth. In some instances, a device may lack a userinterface entirely, but may include display and/or audio outputcapabilities.

Implementations provide a simultaneous, multi-device experience forusers, where such an experience has an enhanced value compared to asingle device interaction. In one example scenario, a user may interactwith three devices including a smartphone, a (e.g., smart) television,and a headless (e.g., no display) personal assistant (PA) device such asthe Amazon Echo™. The user may initiate a session may asking the PAdevice to contact a financial institution to discuss refinancing theuser's home. The PA device may communicate with the financialinstitution, which may join a SR to the session with the user.Initially, the user and the SR may communicate using the PA device as anintermediary that conveys each party's voice communications to theother. The SR may then ask the user to turn on their smart television,and the service may present visual data (e.g., refinance terms or otherinformation) on the television display. The user may use theirsmartphone to enter other information, adjust terms, electronically signa loan agreement, or perform other actions. The various actions taken bythe user through their smartphone, such as adjustment in the terms of aloan agreement, may be reflected in the data that is presented on thetelevision. Through the session, the user and the SR may continuecommunicating verbally through the PA device while using various otherdevices for data output and/or data input.

Traditionally, such a session would be conducted using a single device,such as the user logging into a customer service web site on their homecomputer. However, the single device may have limited capabilities. Byleveraging the various capabilities of the various computing devicesthat may be present in the user's environment, implementations providefor an enriched user experience for each of the individualsparticipating in the session compared to a traditional interaction usinga single device.

FIG. 1 depicts an example system for providing a multi-devicecoordinated user experience, according to implementations of the presentdisclosure. As shown in the example of FIG. 1, one or more users 104 mayinteract with a plurality of client devices 102 in an environment 106.The environment 106 may be a building such as a house or a room within abuilding. The environment 106 may also be at least partly outdoors. Theclient devices 102 may include any suitable type and number of computingdevices. The client devices 102 may include portable (e.g., mobile)computing devices such as a smartphone, tablet computer, wearablecomputer, mobile gaming platform, electronic book reader, and so forth.The client devices 102 may also include less portable computing devicessuch as a desktop computer or laptop computer. The client devices 102may include entertainment devices (e.g., smart televisions, stereosystems, gaming consoles), smart appliances (e.g., heating/coolingsystems, refrigerators, freezers, laundry machines, ovens, etc.), and/orother types of Internet of Things (IoT) devices. The client devices 102may also include any suitable type of personal assistant and/orartificial intelligence (AI) powered device, such as the Amazon Echo™.

The client devices 102 may each include various capabilities for datainput and/or data output. Data input capabilities may include a displayof any suitable type, such as a liquid crystal display (LCD) orlight-emitting diode (LED) display, that is suitable for presenting aninterface such as graphical user interface (GUI), command-lineinterface, or other type of interface, and receiving user input throughthe interface. Data input capabilities may include audio inputcomponent(s) such as one or more microphones configured to receive audio(e.g., speech) input from a user, and software components to analyzeand/or transcribe the audio input, such as speech-to-text (STT)module(s), semantic analysis module(s), voice recognition module(s), andso forth. Data input capabilities may include haptic input component(s)that detect vibrations, rotations, impacts, and/or other movements ofthe client device, such as movements caused by a user. Data inputcapabilities may also include any suitable number of hard controls onthe client devices, such as hardware implemented buttons, knobs, slidercontrols, switches, toggles, keyboards, joysticks, mouse controllers,touchpads, touchscreens, and so forth. Implementations also supportother appropriate types of data input capabilities.

Data output capabilities may include a display of any suitable type,such as a LCD or LED display, that is suitable for presenting a GUI,other types of graphics, images, video content, text data, and so forth.Data output capabilities may include audio output component(s) such asspeakers. Data output capabilities may include haptic outputcomponent(s) such as haptic actuator(s) configured to vibrate and/orotherwise move at least a portion of a client device in response tosignal(s). Data output capabilities may include any suitable numberand/or type of hardware implemented outputs such as LEDs on a clientdevice. Implementations also support other appropriate types of dataoutput capabilities.

In some implementations, different client devices may include differentdata input and/or data output capabilities. For example, FIG. 1 depictsthree different types of client devices 102. The example client device102(1) is a PA device that accepts speech input commands, outputs audiodata, and does not include a display. The example client device 102(2)is a smart television that outputs information through a (e.g., large)display and provides audio output, but may lack a UI or may include a UIwith limited capabilities to data input. The example client device102(3) is a smartphone that may include a (e.g., small) display as wellas audio and haptic output components, and may enable data input througha GUI presented on the display, and/or through speech input.

In some instances, a single user 104 may interact with various clientdevices 102 in an environment 106. In some instances, multiple users 104may interact with various client devices 102 in an environment 106. Insome instances, multiple users 104 may interact with different sets ofclient devices 102 in different environments 106. For example, a firstuser 104(1) may be away from home and a second user 104(2) (e.g., thefirst user's spouse) may be at home. The two users 104 may participatein a same session through the multi-device cooperative user experiencedescribed herein, even though the users 104 may be physically remotefrom one another.

Each of the client devices 102 may execute one or more client modules128, including a client context broker 108, a content exchange module110, and/or a content replication module 112. The client devices 102 mayeach communicate, over one or more networks, with one or more serverdevices 114. The server device(s) 114 may include any appropriate numberand/or type of computing device. The server device(s) 114 may executevarious software module(s) to provide the multi-device cooperative userexperience. The software module(s) may include state managementmodule(s) 116 and context analysis module(s) 118. The server device(s)114 may store session information 120 that describes one or moremulti-device sessions each involving one or more users 104. The serverdevice(s) 114 may also store device information 122 that describes thedata input and/or data output capabilities of various client devices102. In some implementations, one or both of the session information 120and the device information 122 may be stored externally to the serverdevice(s) 114 on storage device(s) that are accessible over one or morenetworks.

In some implementations, the server device(s) 114 may provide variousservice(s) 130 to the users 104. The service(s) 130 may includefinancial service(s) associated with banking, investment management,lending, home financing, and so forth. In some instances, the service(s)130 may include insurance service(s) for health insurance, vehicleinsurance, life insurance, homeowner's insurance, and so forth.Implementations also support other types of service(s) 130. In someimplementations, the multi-device user experience may be provide as partof a multi-device session in which a user 104, such as a servicerepresentative (SR), is providing assistance to one or more other users104 (e.g., customers) with regard to financial services, insuranceservices, and/or other types of services.

In some implementations, a user 104 may register a client device 102 tomake that device available for use during a multi-device session.Registration may include downloading the client module(s) 128 (e.g., asan application or app) onto the client device 102. In someimplementations, the user 104 may authenticate themselves with theservice 130 as part of the registration process using their smartphoneor other client device 102. The service 130 may then send, to the clientdevice 102, a push notification that includes an identifier such as codeand/or customer number. The user 104 may input the identifier to theclient device 102 that is to be registered (e.g., the smartphone or someother client device 102). Based on a verification of the identifier, theservice 130 may register the client device 102 to be available for datainput and/or data output during a multi-device session. The registeredclient device 102 may be associated with the user 104 in the deviceinformation 122. Each client device 102 that is registered for a user104 may add possible data input and/or data output modes for use duringa multi-device session with the user 104.

In some implementations, the service 130 may determine one or more datainput and/or data output capabilities of the client device 102, andstore the determined capabilities of the client device 102 in the deviceinformation 122. In some examples, the client device 102 may indicateits capabilities to the service 130. In some examples, the service 130may receive identification for the client device 102 duringregistration, such as a brand, manufacturer, make, model number, devicetype, version, and so forth. The service 130 may then determine thedevice's capabilities based on the identification information for theclient device 102. For example, the service 130 may access a lookuptable or other data structure that lists the capabilities of variousclient device(s) 102, and retrieve the capabilities of the particularclient device 102 from the data structure. The device information 122may indicate the particular data input and/or data output capabilitiesthat are supported by a particular client device 102. The deviceinformation 122 may also include a more detailed description ofparticular capabilities of the client device 102. For example, if theclient device 102 includes a display for data output, the deviceinformation 122 may include further information regarding the display,such as the type of display (e.g., LED, LCD, etc.), the size of thedisplay (e.g., number of pixel rows and columns, dimensions, aspectratio, etc.), the resolution, the scan type, the supported displayformat(s), and so forth. By storing capability information for eachclient device 102, the device information 122 may indicate which clientdevice(s) 102 may be employed for various kinds of data input and/ordata output during a session.

The client context broker 108 on a client device 102 may collect contextdata 124 that describes a current context of the client device 102. Thecontext data 124 may include information collected by sensor(s) includedin the client device 102, such as environment sensors that collectcontext data 124 regarding the temperature, air pressure, and/or otherenvironmental conditions in proximity to the client device 102. Thecontext data 124 may include a current location of the client device 102that is determined based on signals from a satellite-based positioningsystem such as the Global Positioning System (GPS) or determined usingother techniques. In some instances, the context data 124 may describe acurrent state of the client device 102 such as whether the device iscurrently powered on, active, suspended, asleep, and/or in some otherstate. The context data 124 may also describe the movement and/ororientation of the client device 102, determined based onaccelerometer(s), gyroscopic sensor(s), and/or other types of sensorsincluded in the client device 102.

The collected context data 124 may be communicated to the contextanalysis module(s) 118 on the server device(s) 114. The context analysismodule(s) 118 may analyze the context data 124 to determine a currentcontext of the client device 102 such as its location, the environmentalcontext at the location, whether the device is active, asleep, or insome other state, and/or other context. In some instances, the contextdata 124 may be periodically collected and communicated to the contextanalysis module(s) 118 to enable a determination of the device's contextthat is periodically updated. In some implementations, the clientcontext broker 108 may send an initial set of context data 124 and thensend updated context data 124 may describes the differences and/orchanges compared to the previous set of context data 124. For example,the client context broker 108 may communicate location information ifthe location of the client device 102 changes, but may otherwise notupdate the context analysis module(s) 118 with (e.g., duplicative)location information.

During a multi-device session that includes one or more users 104, aclient device 102 may provide content 126 to be presented on otherclient devices 102. During the session, a client device 102 may receivecontent 126 to be presented on the client device 102. Content 126 mayinclude any suitable type and/or amount of content, including but notlimited to: text, image(s), video content, audio content, graphicscontent (e.g., to be presented in a GUI), haptic content (e.g., to bepresented as a vibration and/or movement of the presenting device), andso forth.

The content exchange module 110 and client replication module 112 mayexecute on a client device 102 to provide content 126 from the clientdevice 102 to one or more other client devices 102. The content exchangemodule 110 may send content 126 from one device to another. For example,content 126 that is input using a GUI on one client device 102 (e.g., atablet computer) may be displayed on another client device 102 (e.g., atelevision). The content replication module 112 may enable content 126to be at least partly replicated on multiple devices. For example, aparticular image, graphic, and/or video may be (e.g., simultaneously)shown on multiple devices such as the user's smartphone, television,game console, and so forth. In such instances, changes made to the dataon one client device 102 may be replicated and shown on the other clientdevice 102. Accordingly, implementations enable a mirroring of datapresented on multiple client device 102. Such mirroring may be updatedin real time, e.g., simultaneously, taking into account the time neededto communicate the data between devices and the processing time on theclient devices 102 and/or server device(s) 114.

The replication of data across multiple client devices 102 may alsoenable multiple, physically remote users 104 to particular in a samemulti-device session. For example, spouses may be physically remote fromone another and able to (e.g., simultaneously) view the same informationregarding a home refinance, with one user 104 viewing the data on theirtelevision and the other user 104 viewing the data on their tabletcomputer. In some instances, a SR may act as another user 104 in such amulti-device, multi-user service session, as described in the example ofFIG. 2.

The state management module(s) 116 may execute on the server device(s)114 to manage the state of various sessions that are currently ongoingor that are in a suspended state. The statement management module(s) 116may store and update session information 120 for each session. Thesession information 120 for a session may describe one or more of thefollowing: the users 104 participating in a session; the client devices102 being used in a session; the current context (or last known context)of each of the client devices 102; the particular data input and/or dataoutput capabilities that are being employed for each client device 102;and/or the particular content 126 that is being presented on variousclient devices 102 through their various data output capabilities. Ininstances where the session is associated with a particular task orprocess, the session information 120 may also describe the current stageor step of the task or process. For example, if the session is to assista user 104 in buying an insurance policy, the session information 120may indicate that the user 104 has provided some of the necessaryinformation but still needs to provide other portions of information tocomplete their application for the policy. In some instances, a user 104may interrupt a session to be continued later (e.g., the next day). Insuch instances, the state management module(s) 116 may use the storedsession information 120 to resume the session in the state where it waspaused, e.g., reconnecting to the various client devices 102 andinstructing the client devices 102 to present the content 126 they werepreviously presenting when the session was interrupted.

In some implementations, the server device(s) 114 may execute a contentformatter 132 that formats the content 126 for presentation onparticular client devices 102 during a session. The content formatter132 may also be described as a data assembler, given that the contentformatter 132 may receive and assemble (e.g., aggregate) data receivedfrom various sources to generate the content 126 to be presented. Thecontent formatter 132 may format content 126 according to the devicecapabilities described in the device information 122. For example,content 126 to be presented on a first client device 102 with a largerdisplay (e.g., a television) may be enriched with additional informationcompared to content 126 that is presented on a second client device 102with a smaller display (e.g., a smartphone), given that expanded displaycapabilities of the first client device 102.

The various communications between the client devices 102 and the serverdevice(s) 114 may be encrypted, compressed, or otherwise secured toensure that the communicated information is not accessed by unauthorizedentities. The session information 120 and/or device information 122 mayalso be encrypted, compressed, or otherwise secured to protect againstunauthorized access.

In some instances the multi-device session may include client devices102 that are associated with different manufacturer “ecosystems”, e.g.,that employ hardware and/or software components that are manufacturedand/or supported by different companies. For example, a multi-devicesession may include one or more client devices 102 that include hardwareand/or software components from Apple™, as well as one or more clientdevices 102 that include hardware and/or software components fromGoogle™. Implementations may enable such diverse client devices to beused in a same multi-device session, in contrast to traditional systemsin which devices from different manufacturers may not be useablecooperatively in a same session.

FIG. 2 depicts an example session 202 conducted through a multi-devicecoordinated user experience, according to implementations of the presentdisclosure. FIG. 2 illustrates an example use case in which a user104(1) (e.g., a customer) is interacting with a user 104(2) (e.g., a SR)to discuss refinancing the customer's house. The users 104 areparticipating in the session 202 through a multi-device user experience,leveraging the various capabilities of multiple client devices 102 toprovide a richer user experience compared to a traditional session thatis conducted using few devices.

In the example of FIG. 2, the user 104(1) requests an initiation of thesession 202 may asking their client device 102(1) (e.g., a headless PAdevice) “should I refinance my house?” The client device 102(1) mayperform a semantic analysis of the question to pick out particularrelevant keywords “refinance” and house.” The client device 102(1) maycommunicate the customer's question as content 126 to the serverdevice(s) 114, which may initiate a service session 202 to assist theuser 104(1). The client device 102(1) and/or server device(s) 114 mayaccess user information that indicates that the user 104(1) conductstheir financial affairs with a particular bank or other financialservice 130. Based on this information, the server device(s) 114 mayforward the user's request to the financial service 130. A user 104(2),e.g., a SR for the service 130, may join the session 202 by sending a(e.g., voice) response “one moment please, I'll run some numbers foryou.” The SR may employ another client device 102(4) to interact withthe server device(s) 114 and to send and/or receive content 126 duringthe session 202. The SR's response may be conveyed through the serverdevice(s) 114 as content 126, and the content 126 may be presented tothe user 104(1) through the audio output capabilities of the clientdevice 102(1). In the example of FIG. 2, the users 104(2) and 104(2) maycontinue communicating verbally to one another during the session 202,using the server device(s) 114 as an intermediary. The server device(s)114 may employ the client device 102(1) as an audio input device toreceive the speech content of the user 104(1) and to play the speechcontent of the user 104(2).

The server device(s) 114 may also employ other client devices 102 toprovide a richer user experience as appropriate during the session 202.For example, the server device(s) 114 may receive context data 124indicating that the user 104(1) has activated another client device102(2), e.g., a television. The server device(s) 114 may provide theuser 104(2) with this updated context, and the user 104(2) may suggestthat the television be used for presenting content 126 (e.g., “can Ishow you the data there?”). If the user 104(1) consents, the user 104(2)may communicate content 126 (e.g., refinance information) to the serverdevice(s) 114 for presentation on the customer's television. The user104(1) may then discuss the presented information with the user 104(2),ask questions, request further information, and so forth. In instanceswhere the user 104(1) asks for a modifications of the information, suchas different terms for the refinance, the user 104(2) may provideupdated information which is displayed on the client device 102(2).

In the example of FIG. 2, the user 104(1) may assent to the terms of therefinance. The user 104(2) may then provide additional content 126 thatis an agreement to be signed to complete the transaction. The user104(2) may provide the additional content 126 to the server device(s)114 and request that it be presented on the client device 102(3), e.g.,the customer's smartphone, based on a determination that the clientdevice 102(3) is currently active and able to receive and presentcontent 126. The user 104(2) may ask the user 104(1) to view andelectronically sign the agreement on the client device 102(3). Thesignature content 126 may then be conveyed to the user 104(2) throughthe server device(s) 114.

Although FIG. 2 presents an example use case involving a single user104(1) interacting with a user 104(2) (e.g., a SR), implementations arenot limited to this example. In some instances, multiple users 104(1)may be participating in the session 202 using different sets of clientdevices 102. In some instances, the users 104(1) may be physicallyremote from one another. For example, one family member may be at homeand participating in the session 202 using the client device 102(1)(e.g., PA device) for audio input and output, using the client device102(2) (e.g., television) to view graphics or other data, and using theclient device 102(3) (e.g., smartphone) to view other data and/orprovide input (e.g., a signature). Another family member may also beparticipating in the session 202 from a remote location, using theirtablet computer for audio input/output, to view graphics (e.g., therefinance information), and to provide input (e.g., their signature)through the GUI of the tablet computer. In such instances, the content126 may be replicated across the various client device(s) 102 for eachof the users 104(1), with the content 126 formatted appropriately forpresentation on the different devices with different displaycapabilities.

In some implementations, the determination of which client device 102 toemploy for data input and/or data output may be based on one or more ofthe following criteria: the particular content 126 to be collected fromand/or presented on the client device(s) 102; the current context of theclient device(s) 102; and/or an indication from the user(s) 104participating in the session 202. The particular content 126 may dictatewhich client device(s) 102 to be used during a session 202. For example,if video content 126 is to be presented to user(s) 104, implementationsmay employ, for the data output, a client device 102 that includes adisplay capable of presenting video content. As another example, if userinput is to be collected from a user 104 (e.g., the user's signature),implementations may employ, for the data input, a client device 102 thatincludes a display capable of presenting a GUI configured to collect theappropriate input. As another example, if audio output is to be providedto user(s) 104, implementations may employ, for the data output, aclient device 102 that includes audio output capabilities (e.g.,speakers and/or appropriate audio processing software). In instanceswhere multiple client devices 102 have the suitable capabilities for aparticular task, multiple client devices 102 may be used (e.g.,simultaneously) for data input and/or data output during a session 202.

In some instances, the current context of the client devices 102registered for a user 104 may be used to determine which clientdevice(s) 102 to use for data input and/or data output. For example,client device(s) 102 may be employed if they are currently in an activestate. In some implementations, if a suitable client device 102 iscurrently inactive (e.g., asleep, suspended), the server device(s) 114may send an activation signal to the client device 102 to activate itfor data input and/or data output. In some instances, a user 104 may beprompted to turn on or otherwise activate a client device 102 so that itmay be used during a session 202. In some examples, location context maybe employed to determine client device(s) 102 to be used. For example,based on the location of the user's smartphone, the service 130 mayinfer that the user 104 is currently at home in an environment 106 withother client device(s) 102 such as a television, PA device, gameconsole, smart appliances, desktop computer, and so forth. Accordingly,such other client device(s) 102 may be used, as appropriate, for datainput and/or data output if the user 104 remains at home during thesession 202. As another example, the service 130 may infer that the user104 is not currently at home and is instead in a hotel. The service maythen suggest that the user 104 turn on their tablet to use the tablet'sdisplay during a session 202. As another example, the service 130 maydetermine that the user 104 is in their car based on a changing locationof the user 104, and the service 130 may employ an automotive computerin the user's car for audio input/output and/or other capabilities.

In some instances, the user 104 may indicate which client device(s) 102are to be used for data input and/or data output. Such an indication maybe made explicitly by the user 104, e.g., through inputs to a UI and/orvoice requests. Such an indication may also be inferred based on otheruser actions. For example, if the user 104 turns on their televisionduring a session 202, an inference may be made that the user 104 wouldlike to use the television for displaying information. The SR may thenask the user 104 if the user 104 would like to use the television fordata output. In some implementations, the client device 102 to be usedfor data input and/or data output may be inferred based on the devicethat the user 104 is currently using, e.g., the device that the user 104employed to request a session 202, authenticate themselves to theservice 130, or perform other actions.

In some implementations, a client device 102 may advertise its presenceto the service 130 and indicate its availability for data input and/ordata output. In some implementations, the service 130 may occasionallyping or otherwise query various client device(s) 102 to determinewhether they are active and available for use. In some implementations,the service 130 may send a signal to wake up and/or otherwise activate aclient device 102 if it is in a suspended state.

FIG. 3 depicts a flow diagram of an example process for registering aclient device 102 for use in a multi-device session, according toimplementations of the present disclosure. Operations of the process maybe performed by one or more of the client module(s) 128, the clientcontext broker 108, the content exchange module 110, the contentreplication module 112, the state management module(s) 116, the contextanalysis module(s) 118, and/or other software module(s) executing on theclient device(s) 102, the server device(s) 114, or elsewhere.

A request may be received (302) to register a new client device 102 fora user 104, as described above. In some implementations, the user 104may be authenticated (304) as an authorized user of the service 130prior to proceeding with registration of the client device 102. If theuser 104 is authenticated, the client device 102 may be registered (306)as described above and associated with the particular user 104. In someimplementations, registration may include providing (308) the clientmodule(s) 128 for installation and execution on the client device 102.The input and/or output capabilities of the client device 102 may bedetermined (310) as described above, and stored (312) in the deviceinformation 122 for the client device 102. Following a successfulregistration of the client device 102 with the service 130, the clientdevice 102 may be employed (according to its capabilities) for datainput and/or data output during a session 202, as described further withreference to FIG. 4.

FIG. 4 depicts a flow diagram of an example process for providing amulti-device coordinated user experience, according to implementationsof the present disclosure. Operations of the process may be performed byone or more of the client module(s) 128, the client context broker 108,the content exchange module 110, the content replication module 112, thestate management module(s) 116, the context analysis module(s) 118,and/or other software module(s) executing on the client device(s) 102,the server device(s) 114, or elsewhere.

A session initiation request may be received (402) from user(s) 104. Insome instances, the user(s) 104 may be authenticated with the service130 prior to initiation of the session 202. A session 202 may beinitiated with the user(s) 104. Session initiation may includeinitializing a portion of the session information 120 for use intracking the state of the session 202. One or more client devices 102may be determined (406) that are associated with the user(s) 104. Insome examples, the client device(s) 102 may be associated with theuser(s) 104 in the device information 122, the client device(s) 102having previously been registered with the service 130.

Context data 124 for the client device(s) 102 may be received andanalyzed (408) to determine a current context for each of the clientdevice(s) 102. The service 130 may determine (410) one or more clientdevice(s) 102 to be used for data input and/or data output during asession 202. As described above, the determination may be based on theparticular types of content 126 to be presented or collected (e.g.,visual data vs. audio data, etc.), the current context of the clientdevice(s) 102, and/or indication(s) from the user(s) 104 in the session202.

In some implementations, the service 130 may decide between multiplepossible client devices 102 to determine an optimal client device 102 touse based on the relative capabilities of the client devices 102. Forexample, a determination may be made that a user 104 has registeredmultiple client devices 102 that include a display capability, such as asmartphone, a tablet computer, and a television. If there is a largequantity of graphical content 126 or video content 126 to be presented,a determination may be made to use the available device with the largestdisplay (e.g., the television). If the content 126 includes text thatmay be difficult to read on a television, a determination may be made touse the tablet computer. In some implementations, the service 130 maytrack preference information for the user 104 and determine clientdevice(s) 102 to use based on the preference information.

For example, preference information collected during previous sessions202 with a user 104 may indicate that the user 104 prefers to view GUIson their tablet computer instead of their smartphone, and prefers tohear audio output through their home stereo system instead of throughtheir PA device. Such preference information may be employed indetermining the client device(s) 102 to use during subsequent sessions202 with the user 104.

The content 126 may be provided (412) for presentation on the clientdevice(s) 102, and/or collected as input to the client device(s) 102. Ifno additional content 126 is to be presented or collected (414), thesession 202 may be terminated (416). If additional content 126 is to bepresented and/or collected (414), the process may return to 408. In thismanner, implementations may monitor device availability and capabilitiesduring a session 202 and dynamically determine how to present content126 and/or collect content 126 from client device(s) 102 as appropriatefor the particular content 126.

In some implementations, the determination of the particular clientdevice(s) 102 to use for presenting and/or collect content 126 during asession 202 may be dynamically determined during the session 202. Asdescribed above, this dynamic determination may be based on theparticular content 126 to be presented and/or collected, the currentcontext of the client device(s) 102 registered to or otherwiseassociated with the users 104 involved in the session 202, and/or theindication(s) received from user(s) 104. In some implementations, adynamic determination of device(s) includes a determination of device(s)that is made during the session 202, and that is not staticallypredetermined prior to an initiation of the session 202. Through dynamicdetermination of input and/or output device(s) to use during a session,implementations are able to dynamically respond as a session 202 (e.g.,data-enriched conversation) develops organically between users 104 inmanner that may not have been predictable prior to initiation of thesession 202. For example, while discussing an auto insurance claim theusers 104 in a session 202 may decide that it would be helpful to viewimage(s) of damage caused to a vehicle during an accident. The SR in thesession 202 may ask that the customer activate their television, and theSR may provide the image(s) to be presented through the display of thetelevision.

FIG. 5 depicts an example computing system, according to implementationsof the present disclosure. The system 500 may be used for any of theoperations described with respect to the various implementationsdiscussed herein. For example, the system 500 may be included, at leastin part, in one or more of the client device(s) 102, the serverdevice(s) 114, and/or other computing device(s) described herein. Thesystem 500 may include one or more processors 510, a memory 520, one ormore storage devices 530, and one or more input/output (I/O) devices 550controllable through one or more I/O interfaces 540. The variouscomponents 510, 520, 530, 540, or 550 may be interconnected through atleast one system bus 560, which may enable the transfer of data betweenthe various modules and components of the system 500.

The processor(s) 510 may be configured to process instructions forexecution within the system 500. The processor(s) 510 may includesingle-threaded processor(s), multi-threaded processor(s), or both. Theprocessor(s) 510 may be configured to process instructions stored in thememory 520 or on the storage device(s) 530. The processor(s) 510 mayinclude hardware-based processor(s) each including one or more cores.The processor(s) 510 may include general purpose processor(s), specialpurpose processor(s), or both.

The memory 520 may store information within the system 500. In someimplementations, the memory 520 includes one or more computer-readablemedia. The memory 520 may include any number of volatile memory units,any number of non-volatile memory units, or both volatile andnon-volatile memory units. The memory 520 may include read-only memory,random access memory, or both. In some examples, the memory 520 may beemployed as active or physical memory by one or more executing softwaremodules.

The storage device(s) 530 may be configured to provide (e.g.,persistent) mass storage for the system 500. In some implementations,the storage device(s) 530 may include one or more computer-readablemedia. For example, the storage device(s) 530 may include a floppy diskdevice, a hard disk device, an optical disk device, or a tape device.The storage device(s) 530 may include read-only memory, random accessmemory, or both. The storage device(s) 530 may include one or more of aninternal hard drive, an external hard drive, or a removable drive.

One or both of the memory 520 or the storage device(s) 530 may includeone or more computer-readable storage media (CRSM). The CRSM may includeone or more of an electronic storage medium, a magnetic storage medium,an optical storage medium, a magneto-optical storage medium, a quantumstorage medium, a mechanical computer storage medium, and so forth. TheCRSM may provide storage of computer-readable instructions describingdata structures, processes, applications, programs, other modules, orother data for the operation of the system 500. In some implementations,the CRSM may include a data store that provides storage ofcomputer-readable instructions or other information in a non-transitoryformat. The CRSM may be incorporated into the system 500 or may beexternal with respect to the system 500. The CRSM may include read-onlymemory, random access memory, or both. One or more CRSM suitable fortangibly embodying computer program instructions and data may includeany type of non-volatile memory, including but not limited to:semiconductor memory devices, such as EPROM, EEPROM, and flash memorydevices; magnetic disks such as internal hard disks and removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks. In some examples,the processor(s) 510 and the memory 520 may be supplemented by, orincorporated into, one or more application-specific integrated circuits(ASICs).

The system 500 may include one or more I/O devices 550. The I/Odevice(s) 550 may include one or more input devices such as a keyboard,a mouse, a pen, a game controller, a touch input device, an audio inputdevice (e.g., a microphone), a gestural input device, a haptic inputdevice, an image or video capture device (e.g., a camera), or otherdevices. In some examples, the I/O device(s) 550 may also include one ormore output devices such as a display, LED(s), an audio output device(e.g., a speaker), a printer, a haptic output device, and so forth. TheI/O device(s) 550 may be physically incorporated in one or morecomputing devices of the system 500, or may be external with respect toone or more computing devices of the system 500.

The system 500 may include one or more I/O interfaces 540 to enablecomponents or modules of the system 500 to control, interface with, orotherwise communicate with the I/O device(s) 550. The I/O interface(s)540 may enable information to be transferred in or out of the system500, or between components of the system 500, through serialcommunication, parallel communication, or other types of communication.For example, the I/O interface(s) 540 may comply with a version of theRS-232 standard for serial ports, or with a version of the IEEE 1284standard for parallel ports. As another example, the I/O interface(s)540 may be configured to provide a connection over Universal Serial Bus(USB) or Ethernet. In some examples, the I/O interface(s) 540 may beconfigured to provide a serial connection that is compliant with aversion of the IEEE 1394 standard.

The I/O interface(s) 540 may also include one or more network interfacesthat enable communications between computing devices in the system 500,or between the system 500 and other network-connected computing systems.The network interface(s) may include one or more network interfacecontrollers (NICs) or other types of transceiver devices configured tosend and receive communications over one or more networks using anynetwork protocol.

Computing devices of the system 500 may communicate with one another, orwith other computing devices, using one or more networks. Such networksmay include public networks such as the internet, private networks suchas an institutional or personal intranet, or any combination of privateand public networks. The networks may include any type of wired orwireless network, including but not limited to local area networks(LANs), wide area networks (WANs), wireless WANs (WWANs), wireless LANs(WLANs), mobile communications networks (e.g., 3G, 4G, Edge, etc.), andso forth. In some implementations, the communications between computingdevices may be encrypted or otherwise secured. For example,communications may employ one or more public or private cryptographickeys, ciphers, digital certificates, or other credentials supported by asecurity protocol, such as any version of the Secure Sockets Layer (SSL)or the Transport Layer Security (TLS) protocol.

The system 500 may include any number of computing devices of any type.The computing device(s) may include, but are not limited to: a personalcomputer, a smartphone, a tablet computer, a wearable computer, animplanted computer, a mobile gaming device, an electronic book reader,an automotive computer, a desktop computer, a laptop computer, anotebook computer, a game console, a home entertainment device, anetwork computer, a server computer, a mainframe computer, a distributedcomputing device (e.g., a cloud computing device), a microcomputer, asystem on a chip (SoC), a system in a package (SiP), and so forth.Although examples herein may describe computing device(s) as physicaldevice(s), implementations are not so limited. In some examples, acomputing device may include one or more of a virtual computingenvironment, a hypervisor, an emulation, or a virtual machine executingon one or more physical computing devices. In some examples, two or morecomputing devices may include a cluster, cloud, farm, or other groupingof multiple devices that coordinate operations to provide loadbalancing, failover support, parallel processing capabilities, sharedstorage resources, shared networking capabilities, or other aspects.

Implementations and all of the functional operations described in thisspecification may be realized in digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Implementations may be realized asone or more computer program products, i.e., one or more modules ofcomputer program instructions encoded on a computer readable medium forexecution by, or to control the operation of, data processing apparatus.The computer readable medium may be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter effecting a machine-readable propagated signal, or a combinationof one or more of them. The term “computing system” encompasses allapparatus, devices, and machines for processing data, including by wayof example a programmable processor, a computer, or multiple processorsor computers. The apparatus may include, in addition to hardware, codethat creates an execution environment for the computer program inquestion, e.g., code that constitutes processor firmware, a protocolstack, a database management system, an operating system, or acombination of one or more of them. A propagated signal is anartificially generated signal, e.g., a machine-generated electrical,optical, or electromagnetic signal that is generated to encodeinformation for transmission to suitable receiver apparatus.

A computer program (also known as a program, software, softwareapplication, script, or code) may be written in any appropriate form ofprogramming language, including compiled or interpreted languages, andit may be deployed in any appropriate form, including as a standaloneprogram or as a module, component, subroutine, or other unit suitablefor use in a computing environment. A computer program does notnecessarily correspond to a file in a file system. A program may bestored in a portion of a file that holds other programs or data (e.g.,one or more scripts stored in a markup language document), in a singlefile dedicated to the program in question, or in multiple coordinatedfiles (e.g., files that store one or more modules, sub programs, orportions of code). A computer program may be deployed to be executed onone computer or on multiple computers that are located at one site ordistributed across multiple sites and interconnected by a communicationnetwork.

The processes and logic flows described in this specification may beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows may also be performedby, and apparatus may also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any appropriate kind of digital computer.Generally, a processor may receive instructions and data from a readonly memory or a random access memory or both. Elements of a computercan include a processor for performing instructions and one or morememory devices for storing instructions and data. Generally, a computermay also include, or be operatively coupled to receive data from ortransfer data to, or both, one or more mass storage devices for storingdata, e.g., magnetic, magneto optical disks, or optical disks. However,a computer need not have such devices. Moreover, a computer may beembedded in another device, e.g., a mobile telephone, a personal digitalassistant (PDA), a mobile audio player, a Global Positioning System(GPS) receiver, to name just a few. Computer readable media suitable forstoring computer program instructions and data include all forms ofnon-volatile memory, media and memory devices, including by way ofexample semiconductor memory devices, e.g., EPROM, EEPROM, and flashmemory devices; magnetic disks, e.g., internal hard disks or removabledisks; magneto optical disks; and CD ROM and DVD-ROM disks. Theprocessor and the memory may be supplemented by, or incorporated in,special purpose logic circuitry.

To provide for interaction with a user, implementations may be realizedon a computer having a display device, e.g., a CRT (cathode ray tube) orLCD (liquid crystal display) monitor, for displaying information to theuser and a keyboard and a pointing device, e.g., a mouse or a trackball,by which the user may provide input to the computer. Other kinds ofdevices may be used to provide for interaction with a user as well; forexample, feedback provided to the user may be any appropriate form ofsensory feedback, e.g., visual feedback, auditory feedback, or tactilefeedback; and input from the user may be received in any appropriateform, including acoustic, speech, or tactile input.

Implementations may be realized in a computing system that includes aback end component, e.g., as a data server, or that includes amiddleware component, e.g., an application server, or that includes afront end component, e.g., a client computer having a graphical UI or aweb browser through which a user may interact with an implementation, orany appropriate combination of one or more such back end, middleware, orfront end components. The components of the system may be interconnectedby any appropriate form or medium of digital data communication, e.g., acommunication network. Examples of communication networks include alocal area network (“LAN”) and a wide area network (“WAN”), e.g., theInternet.

The computing system may include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of the disclosure or of what maybe claimed, but rather as descriptions of features specific toparticular implementations. Certain features that are described in thisspecification in the context of separate implementations may also beimplemented in combination in a single implementation. Conversely,various features that are described in the context of a singleimplementation may also be implemented in multiple implementationsseparately or in any suitable sub-combination. Moreover, althoughfeatures may be described above as acting in certain combinations andeven initially claimed as such, one or more features from a claimedcombination may in some examples be excised from the combination, andthe claimed combination may be directed to a sub-combination orvariation of a sub-combination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemsmay generally be integrated together in a single software product orpackaged into multiple software products.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. For example, various formsof the flows shown above may be used, with steps re-ordered, added, orremoved. Accordingly, other implementations are within the scope of thefollowing claims.

The invention claimed is:
 1. A computer-implemented method performed byat least one processor, the method comprising: determining, by the atleast one processor, a first client device to be used for collectingfirst content during a communication session between the first clientdevice and a service representative (SR) device, the first client devicebeing determined during the communication session based at least partlyon the first client device including a particular data input capabilityfor collecting the first content; receiving, by the at least oneprocessor, the first content that is collected through the first clientdevice during the communication session; determining, by the at leastone processor, a second client device to be used for presenting secondcontent during the communication session, the second client device beingdetermined during the communication session based at least partly on thesecond client device including a particular data output capability forpresenting the second content, the second client device being differentthan the first client device; providing, by the at least one processor,the second content for presentation through the second client deviceduring the communication session; storing session information of thecommunication session, the session information indicating that the firstclient device is being used for collecting the first content and thesecond client device is being used for presenting the second content;and reestablishing, by accessing the stored session information and uponan interruption of the communication session, the communication sessionby continuing the use of the first client device for collecting thefirst content and continuing the use of the second client device forpresenting the second content.
 2. The method of claim 1, wherein: thefirst content is provided through the first client device by a firstuser participating in the communication session; the second content ispresented through the second client device to a second userparticipating in the communication session; and the first client deviceis in a different geographic location from the second client device. 3.The method of claim 1, wherein the determining of the second clientdevice is further based on a current context of the second clientdevice, including one or more of a location or a state of the secondclient device.
 4. The method of claim 1, wherein the determining of thesecond client device is further based on an indication of the SR usingthe SR device.
 5. The method of claim 1, wherein: the first contentincludes audio input provided by a first user participating in thecommunication session; the second content includes visual outputprovided by a second user participating in the communication session;and the second content is determined based at least partly on the firstcontent.
 6. The method of claim 1, further comprising: determining, bythe at least one processor, a third client device to be used forpresenting third content during the communication session, the thirdclient device being determined during the communication session based atleast partly on the third client device including a particular dataoutput capability for presenting the third content, the third clientdevice being different than the first client device and the secondclient device; and providing, by the at least one processor, the thirdcontent for presentation through the third client device during thecommunication session.
 7. The method of claim 6, wherein at least one ofthe second content or the third content is replicated for presentationthrough both the second client device and the third client device duringthe communication session.
 8. The method of claim 1, further comprising:registering, by the at least one processor, the first client device andthe second client device, the registering including determining andstoring a set of capabilities of each of the first client device and thesecond client device, the set of capabilities including one or more of adata output capability or a data input capability.
 9. A system,comprising: at least one processor; and a memory communicatively coupledto the at least one processor, the memory storing instructions which,when executed by the at least one processor, cause the at least oneprocessor to perform operations comprising: determining a first clientdevice to be used for collecting first content during a communicationsession between the first client device and a service representative(SR) device, the first client device being determined during thecommunication session based at least partly on the first client deviceincluding a particular data input capability for collecting the firstcontent; receiving the first content that is collected through the firstclient device during the communication session; determining a secondclient device to be used for presenting second content during thecommunication session, the second client device being determined duringthe communication session based at least partly on the second clientdevice including a particular data output capability for presenting thesecond content, the second client device being different than the firstclient device; providing the second content for presentation through thesecond client device during the communication session; storing sessioninformation of the communication session, the session informationindicating that the first client device is being used for collecting thefirst content and the second client device is being used for presentingthe second content; and reestablishing, by accessing the stored sessioninformation and upon an interruption of the communication session, thecommunication session by continuing the use of the first client devicefor collecting the first content and continuing the use of the secondclient device for presenting the second content.
 10. The system of claim9, wherein: the first content is provided through the first clientdevice by a first user participating in the communication session; thesecond content is presented through the second client device to a seconduser participating in the communication session; and the first clientdevice is in a different geographic location from the second clientdevice.
 11. The system of claim 9, wherein the determining of the secondclient device is further based on a current context of the second clientdevice, including one or more of a location or a state of the secondclient device.
 12. The system of claim 9, wherein the determining of thesecond client device is further based on an indication of the SR usingthe SR device.
 13. The system of claim 9, wherein: the first contentincludes audio input provided by a first user participating in thecommunication session; the second content includes visual outputprovided by a second user participating in the communication session;and the second content is determined based at least partly on the firstcontent.
 14. The system of claim 9, the operations further comprising:determining a third client device to be used for presenting thirdcontent during the communication session, the third client device beingdetermined during the communication session based at least partly on thethird client device including a particular data output capability forpresenting the third content, the third client device being differentthan the first client device and the second client device; and providingthe third content for presentation through the third client deviceduring the communication session.
 15. One or more non-transitorycomputer-readable media storing instructions which, when executed by atleast one processor, cause the at least one processor to performoperations comprising: determining a first client device to be used forcollecting first content during a communication session between thefirst client device and a service representative (SR) device, the firstclient device being determined during the communication session based atleast partly on the first client device including a particular datainput capability for collecting the first content; receiving the firstcontent that is collected through the first client device during thecommunication session; determining a second client device to be used forpresenting second content during the communication session, the secondclient device being determined during the communication session based atleast partly on the second client device including a particular dataoutput capability for presenting the second content, the second clientdevice being different than the first client device; providing thesecond content for presentation through the second client device duringthe communication session; storing session information of thecommunication session, the session information indicating that the firstclient device is being used for collecting the first content and thesecond client device is being used for presenting the second content;and reestablishing, by accessing the stored session information and uponan interruption of the communication session, the communication sessionby continuing the use of the first client device for collecting thefirst content and continuing the use of the second client device forpresenting the second content.
 16. The one or more non-transitorycomputer-readable media of claim 15, wherein the determining of thesecond client device is further based on an indication of the SR usingthe SR device.
 17. The one or more non-transitory computer-readablemedia of claim 15, wherein: the first content includes audio inputprovided by a first user participating in the communication session; thesecond content includes visual output provided by a second userparticipating in the communication session; and the second content isdetermined based at least partly on the first content.
 18. The one ormore non-transitory computer-readable media of claim 15, the operationsfurther comprising: determining a third client device to be used forpresenting third content during the communication session, the thirdclient device being determined during the communication session based atleast partly on the third client device including a particular dataoutput capability for presenting the third content, the third clientdevice being different than the first client device and the secondclient device; and providing the third content for presentation throughthe third client device during the communication session.
 19. The one ormore non-transitory computer-readable media of claim 18, wherein atleast one of the second content or the third content is replicated forpresentation through both the second client device and the third clientdevice during the communication session.
 20. The one or morenon-transitory computer-readable media of claim 15, the operationsfurther comprising: registering the first client device and the secondclient device, the registering including determining and storing a setof capabilities of each of the first client device and the second clientdevice, the set of capabilities including one or more of a data outputcapability or a data input capability.